Con duct ls #224

asmacdo · 2024-11-12T04:21:19Z

Fixes #185

summaries

Command: sleep 0.11
	Wall Clock Time: 0.112 sec
	Memory Peak Usage (RSS): 2.0 MB
	Virtual Memory Peak Usage (VSZ): 232.4 MB
	Memory Peak Percentage: 0.00%
	CPU Peak Usage: 0.00%

Command: sleep 0.1
	Wall Clock Time: 0.102 sec
	Memory Peak Usage (RSS): 2.0 MB
	Virtual Memory Peak Usage (VSZ): 232.4 MB
	Memory Peak Percentage: 0.00%
	CPU Peak Usage: 0.00%

yaml

command: sleep 0.11
duct_version: 0.8.0
env: {}
execution_summary:
  average_pcpu: 0.0
  average_pmem: 0.0
  average_rss: 1961984
  average_vsz: 232374272
  command: sleep 0.11
  end_time: 1731385053.3107307
  exit_code: 0
  logs_prefix: .duct/logs/2024.11.11T22.17.33-130359_
  num_reports: 2
  num_samples: 1
  peak_pcpu: 0.0
  peak_pmem: 0.0
  peak_rss: 1961984
  peak_vsz: 232374272
  start_time: 1731385053.198443
  wall_clock_time: 0.11228775978088379
gpu: null
output_paths:
  info: .duct/logs/2024.11.11T22.17.33-130359_info.json
  prefix: .duct/logs/2024.11.11T22.17.33-130359_
  stderr: .duct/logs/2024.11.11T22.17.33-130359_stderr
  stdout: .duct/logs/2024.11.11T22.17.33-130359_stdout
  usage: .duct/logs/2024.11.11T22.17.33-130359_usage.json
schema_version: 0.2.0
system:
  cpu_total: 20
  hostname: fancy
  memory_total: 33336127488
  uid: 1000
  user: austin

command: sleep 0.1
duct_version: 0.8.0
env: {}
execution_summary:
  average_pcpu: 0.0
  average_pmem: 0.0
  average_rss: 1978368
  average_vsz: 232374272
  command: sleep 0.1
  end_time: 1731385056.415762
  exit_code: 0
  logs_prefix: .duct/logs/2024.11.11T22.17.36-130380_
  num_reports: 2
  num_samples: 1
  peak_pcpu: 0.0
  peak_pmem: 0.0
  peak_rss: 1978368
  peak_vsz: 232374272
  start_time: 1731385056.3133166
  wall_clock_time: 0.10244536399841309
gpu: null
output_paths:
  info: .duct/logs/2024.11.11T22.17.36-130380_info.json
  prefix: .duct/logs/2024.11.11T22.17.36-130380_
  stderr: .duct/logs/2024.11.11T22.17.36-130380_stderr
  stdout: .duct/logs/2024.11.11T22.17.36-130380_stdout
  usage: .duct/logs/2024.11.11T22.17.36-130380_usage.json
schema_version: 0.2.0
system:
  cpu_total: 20
  hostname: fancy
  memory_total: 33336127488
  uid: 1000
  user: austin

json

{"command": "sleep 0.11", "system": {"cpu_total": 20, "memory_total": 33336127488, "hostname": "fancy", "uid": 1000, "user": "austin"}, "env": {}, "gpu": null, "duct_version": "0.8.0", "schema_version": "0.2.0", "execution_summary": {"exit_code": 0, "command": "sleep 0.11", "logs_prefix": ".duct/logs/2024.11.11T22.17.33-130359_", "wall_clock_time": 0.11228775978088379, "peak_rss": 1961984, "average_rss": 1961984, "peak_vsz": 232374272, "average_vsz": 232374272, "peak_pmem": 0.0, "average_pmem": 0.0, "peak_pcpu": 0.0, "average_pcpu": 0.0, "num_samples": 1, "num_reports": 2, "start_time": 1731385053.198443, "end_time": 1731385053.3107307}, "output_paths": {"stdout": ".duct/logs/2024.11.11T22.17.33-130359_stdout", "stderr": ".duct/logs/2024.11.11T22.17.33-130359_stderr", "usage": ".duct/logs/2024.11.11T22.17.33-130359_usage.json", "info": ".duct/logs/2024.11.11T22.17.33-130359_info.json", "prefix": ".duct/logs/2024.11.11T22.17.33-130359_"}}
{"command": "sleep 0.1", "system": {"cpu_total": 20, "memory_total": 33336127488, "hostname": "fancy", "uid": 1000, "user": "austin"}, "env": {}, "gpu": null, "duct_version": "0.8.0", "schema_version": "0.2.0", "execution_summary": {"exit_code": 0, "command": "sleep 0.1", "logs_prefix": ".duct/logs/2024.11.11T22.17.36-130380_", "wall_clock_time": 0.10244536399841309, "peak_rss": 1978368, "average_rss": 1978368, "peak_vsz": 232374272, "average_vsz": 232374272, "peak_pmem": 0.0, "average_pmem": 0.0, "peak_pcpu": 0.0, "average_pcpu": 0.0, "num_samples": 1, "num_reports": 2, "start_time": 1731385056.3133166, "end_time": 1731385056.415762}, "output_paths": {"stdout": ".duct/logs/2024.11.11T22.17.36-130380_stdout", "stderr": ".duct/logs/2024.11.11T22.17.36-130380_stderr", "usage": ".duct/logs/2024.11.11T22.17.36-130380_usage.json", "info": ".duct/logs/2024.11.11T22.17.36-130380_info.json", "prefix": ".duct/logs/2024.11.11T22.17.36-130380_"}}

jsonpp

{'command': 'sleep 0.11',
 'duct_version': '0.8.0',
 'env': {},
 'execution_summary': {'average_pcpu': 0.0,
                       'average_pmem': 0.0,
                       'average_rss': 1961984,
                       'average_vsz': 232374272,
                       'command': 'sleep 0.11',
                       'end_time': 1731385053.3107307,
                       'exit_code': 0,
                       'logs_prefix': '.duct/logs/2024.11.11T22.17.33-130359_',
                       'num_reports': 2,
                       'num_samples': 1,
                       'peak_pcpu': 0.0,
                       'peak_pmem': 0.0,
                       'peak_rss': 1961984,
                       'peak_vsz': 232374272,
                       'start_time': 1731385053.198443,
                       'wall_clock_time': 0.11228775978088379},
 'gpu': None,
 'output_paths': {'info': '.duct/logs/2024.11.11T22.17.33-130359_info.json',
                  'prefix': '.duct/logs/2024.11.11T22.17.33-130359_',
                  'stderr': '.duct/logs/2024.11.11T22.17.33-130359_stderr',
                  'stdout': '.duct/logs/2024.11.11T22.17.33-130359_stdout',
                  'usage': '.duct/logs/2024.11.11T22.17.33-130359_usage.json'},
 'schema_version': '0.2.0',
 'system': {'cpu_total': 20,
            'hostname': 'fancy',
            'memory_total': 33336127488,
            'uid': 1000,
            'user': 'austin'}}
{'command': 'sleep 0.1',
 'duct_version': '0.8.0',
 'env': {},
 'execution_summary': {'average_pcpu': 0.0,
                       'average_pmem': 0.0,
                       'average_rss': 1978368,
                       'average_vsz': 232374272,
                       'command': 'sleep 0.1',
                       'end_time': 1731385056.415762,
                       'exit_code': 0,
                       'logs_prefix': '.duct/logs/2024.11.11T22.17.36-130380_',
                       'num_reports': 2,
                       'num_samples': 1,
                       'peak_pcpu': 0.0,
                       'peak_pmem': 0.0,
                       'peak_rss': 1978368,
                       'peak_vsz': 232374272,
                       'start_time': 1731385056.3133166,
                       'wall_clock_time': 0.10244536399841309},
 'gpu': None,
 'output_paths': {'info': '.duct/logs/2024.11.11T22.17.36-130380_info.json',
                  'prefix': '.duct/logs/2024.11.11T22.17.36-130380_',
                  'stderr': '.duct/logs/2024.11.11T22.17.36-130380_stderr',
                  'stdout': '.duct/logs/2024.11.11T22.17.36-130380_stdout',
                  'usage': '.duct/logs/2024.11.11T22.17.36-130380_usage.json'},
 'schema_version': '0.2.0',
 'system': {'cpu_total': 20,
            'hostname': 'fancy',
            'memory_total': 33336127488,
            'uid': 1000,
            'user': 'austin'}}

TODO

pyout

src/con_duct/suite/ls.py

codecov · 2024-11-12T17:25:37Z

Codecov Report

Attention: Patch coverage is 41.17647% with 20 lines in your changes missing coverage. Please review.

Project coverage is 92.66%. Comparing base (50a711e) to head (49aed82).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
src/con_duct/suite/ls.py	31.03%	20 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #224      +/-   ##
==========================================
- Coverage   95.36%   92.66%   -2.71%     
==========================================
  Files           4        5       +1     
  Lines         626      668      +42     
  Branches       70       78       +8     
==========================================
+ Hits          597      619      +22     
- Misses         15       35      +20     
  Partials       14       14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

=== Do not change lines below === { "chain": [], "cmd": "./.update-readme-help.py", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

src/con_duct/suite/ls.py

asmacdo · 2025-01-15T16:40:02Z

Overall:

ls should be more of a list, rather than a list of long summaries.
maybe pyout will be cleaner

Info that seems most important:

command
info.json path
exit code
walltime

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

yarikoptic · 2025-01-15T18:27:34Z

note that command might be too long to show in full "by default", max memory resource i think would also be nice to know.

asmacdo · 2025-01-15T18:45:00Z

@yarikoptic i've hacked together something with pyout

I'm not sure how I want to handle the "include_only" list, since ideally that would be agnostic to the output format, and I don't see a clean way to incorporate with LS_SUMMARY_FORMAT. But pyout is nice and easy!

yarikoptic · 2025-01-15T20:58:46Z

I'm not sure how I want to handle the "include_only" list,

sorry -- could you describe what you mean by "include_only" list here?

asmacdo · 2025-01-15T21:54:57Z

@yarikoptic https://github.com/con/duct/pull/224/files#diff-150598b33b1982d47d2bc3f78f8ba2333963da1d5ebe2b6211cba03b7c259cfcR64

Restricting the columns to add to the table

asmacdo added 7 commits November 11, 2024 15:20

Initial implementation of con-duct ls

5108f51

Add summaries

bfc90e3

Fixup remove redundant, included in argparse choices

1ce361c

Add json and jsonpp to con-duct ls

2d6b05a

add yaml format to ls

7e07953

Do jsonpp as yarik suggested

c73a990

fixup rm reference comments

99e2b9f

yarikoptic reviewed Nov 12, 2024

View reviewed changes

src/con_duct/suite/ls.py Outdated Show resolved Hide resolved

yarikoptic reviewed Nov 12, 2024

View reviewed changes

src/con_duct/suite/ls.py Outdated Show resolved Hide resolved

yarikoptic reviewed Nov 12, 2024

View reviewed changes

src/con_duct/suite/ls.py Outdated Show resolved Hide resolved

yarikoptic reviewed Nov 12, 2024

View reviewed changes

src/con_duct/suite/ls.py Show resolved Hide resolved

yarikoptic reviewed Nov 12, 2024

View reviewed changes

src/con_duct/suite/ls.py Outdated Show resolved Hide resolved

asmacdo added 3 commits November 12, 2024 11:14

RF: Dry up info file loading

fd4135a

Dry up return values, and handle invalid choice situation

08e8afb

Add yaml dep for con-duct ls

594e86a

asmacdo added 2 commits November 12, 2024 11:26

use camel case

e1ffad9

[DATALAD RUNCMD] Autoupdate readme

38bf3de

=== Do not change lines below === { "chain": [], "cmd": "./.update-readme-help.py", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

asmacdo added the semver-minor Increment the minor version when merged label Nov 12, 2024

asmacdo mentioned this pull request Nov 12, 2024

Add datetime of command start to info.json #221

Open

bf: fix error when not passing yaml

969ecb6

yarikoptic reviewed Dec 3, 2024

View reviewed changes

src/con_duct/suite/ls.py Outdated Show resolved Hide resolved

asmacdo and others added 4 commits January 15, 2025 11:44

Update src/con_duct/suite/ls.py

49aed82

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

Use elifs

5bde53c

Cleanup import and type annotation

e8c0465

Fixup: type annotations

8a51694

WIP: Hack solution with pyout

e64417f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Con duct ls #224

Con duct ls #224

asmacdo commented Nov 12, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading

asmacdo commented Jan 15, 2025

yarikoptic commented Jan 15, 2025

asmacdo commented Jan 15, 2025

yarikoptic commented Jan 15, 2025

asmacdo commented Jan 15, 2025

Con duct ls #224

Are you sure you want to change the base?

Con duct ls #224

Conversation

asmacdo commented Nov 12, 2024 • edited Loading

summaries

yaml

json

jsonpp

TODO

codecov bot commented Nov 12, 2024 • edited Loading

Codecov Report

asmacdo commented Jan 15, 2025

yarikoptic commented Jan 15, 2025

asmacdo commented Jan 15, 2025

yarikoptic commented Jan 15, 2025

asmacdo commented Jan 15, 2025

asmacdo commented Nov 12, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading